Loop Estimator for Discounted Values in Markov Reward Processes

At the working heart of policy iteration algorithms commonly used and studied in discounted setting reinforcement learning, evaluation step estimates value states with samples from a Markov reward process induced by following decision process. We propose simple efficient estimator called loop that exploits regenerative structure processes without explicitly estimating full model. Our method enjoys space complexity O(1) when single positive recurrent state s unlike TD O(S) or model-based methods O(S^2). Moreover, enables us to show, relying on generative model approach, has an instance-dependent convergence rate O~(\sqrt{\tau_s/T}) over steps T sample path, where \tau_s is maximal expected hitting time s. In preliminary numerical experiments, outperforms model-free methods, such as TD(k), competitive estimator.